Perceptual uncertainty explains activation differences between audiovisual congruent speech and McGurk stimuli.

Face-to-face communication relies on the integration of acoustic speech signals with the corresponding facial articulations. In the McGurk illusion, an auditory /ba/ phoneme presented simultaneously with a facial articulation of a /ga/ (i.e., viseme), is typically fused into an illusory 'da' percept. Despite its widespread use as an index of audiovisual speech integration, critics argue that it arises from perceptual processes that differ categorically from natural speech recognition. Conversely, Bayesian theoretical frameworks suggest that both the illusory McGurk and the veridical audiovisual congruent speech percepts result from probabilistic inference based on noisy sensory signals. According to these models, the inter-sensory conflict in McGurk stimuli may only increase observers' perceptual uncertainty. This functional magnetic resonance imaging (fMRI) study presented participants (20 male and 24 female) with audiovisual congruent, McGurk (i.e., auditory /ba/ + visual /ga/), and incongruent (i.e., auditory /ga/ + visual /ba/) stimuli along with their unisensory counterparts in a syllable categorization task. Behaviorally, observers' response entropy was greater for McGurk compared to congruent audiovisual stimuli. At the neural level, McGurk stimuli increased activations in a widespread neural system, extending from the inferior frontal sulci (IFS) to the pre-supplementary motor area (pre-SMA) and insulae, typically involved in cognitive control processes. Crucially, in line with Bayesian theories these activation increases were fully accounted for by observers' perceptual uncertainty as measured by their response entropy. Our findings suggest that McGurk and congruent speech processing rely on shared neural mechanisms, thereby supporting the McGurk illusion as a valid measure of natural audiovisual speech perception.

Parafoveal and foveal N400 effects in natural reading: A timeline of semantic processing from fixation-related potentials.

The depth at which parafoveal words are processed during reading is an ongoing topic of debate. Recent studies using RSVP-with-flanker paradigms have shown that implausible words within sentences elicit an N400 component while they are still in parafoveal vision, suggesting that the semantics of parafoveal words can be accessed to rapidly update the sentence representation. To study this effect in natural reading, we combined the coregistration of eye movements and EEG with the deconvolution modeling of fixation-related potentials (FRPs) to test whether semantic plausibility is processed parafoveally during Chinese sentence reading. For one target word per sentence, both its parafoveal and foveal plausibility were orthogonally manipulated using the boundary paradigm. Consistent with previous eye movement studies, we observed a delayed effect of parafoveal plausibility on fixation durations that only emerged on the foveal word. Crucially, in FRPs aligned to the pretarget fixation, a clear N400 effect emerged already based on parafoveal plausibility, with more negative voltages for implausible previews. Once participants fixated the target, we again observed an N400 effect of foveal plausibility. Interestingly, this foveal N400 was absent whenever the preview had been implausible, indicating that when a word's (im)plausibility is already processed in parafoveal vision, this information is not revised anymore upon direct fixation. Implausible words also elicited a late positive component (LPC), but exclusively when in foveal vision. Our results not only provide convergent neural and behavioral evidence for the parafoveal uptake of semantic information, but also indicate different contributions of parafoveal versus foveal information toward higher level sentence processing.

Atypical brain lateralization for speech processing at the sublexical level in autistic children revealed by fNIRS.

Autistic children often exhibit atypical brain lateralization of language processing, but it is unclear what aspects of language contribute to this phenomenon. This study employed functional near-infrared spectroscopy to measure hemispheric lateralization by estimating hemodynamic responses associated with processing linguistic and non-linguistic auditory stimuli. The study involved a group of autistic children (N = 20, mean age = 5.8 years) and a comparison group of nonautistic peers (N = 20, mean age = 6.5 years). The children were presented with stimuli with systematically decreasing linguistic relevance: naturalistic native speech, meaningless native speech with scrambled word order, nonnative speech, and music. The results revealed that both groups showed left lateralization in the temporal lobe when listening to naturalistic native speech. However, the distinction emerged between autism and nonautistic in terms of processing the linguistic hierarchy. Specifically, the nonautistic comparison group demonstrated a systematic reduction in left lateralization as linguistic relevance decreased. In contrast, the autism group displayed no such pattern and showed no lateralization when listening to scrambled native speech accompanied by enhanced response in the right hemisphere. These results provide evidence of atypical neural specialization for spoken language in preschool- and school-age autistic children and shed new light on the underlying linguistic correlates contributing to such atypicality at the sublexical level.

Comparing word recognition in simplified and traditional Chinese: A megastudy approach.

Although it is well established that the visual complexity of a written word can influence processing, it is far less clear from a cross-script perspective, whether the overall visual complexity of the entire written lexicon also affects word recognition. This question can be answered with the data in megastudy of lexical decision in Chinese (MELD-CH), which was developed with over 800 participants responding to 12,587 simplified and traditional Chinese words. The results showed that lexical decision was slower but more accurate in simplified Chinese, which has about 22.5% less strokes, than traditional Chinese. This pattern could not be explained by a speed-accuracy trade-off. Moderate correlations were found in response times and error rates between the two scripts, indicating considerable overlap in processing despite the script difference. In addition, (generalised) linear mixed-effects modelling was used to examine whether the simplified and traditional Chinese groups differed in sensitivity towards linguistic variables. The results showed that the effects of word frequency, word length, and number of strokes were stronger in recognising simplified Chinese words, whereas the effects of number of words formed and number of meanings of the constituent characters were stronger in traditional Chinese. These results suggested that the visual-perceptual processing demands of simplified Chinese might force readers to focus more on local properties of the words, making them less sensitive to global properties that are defined over the entire lexicon. Finally, limitations and alternative explanations of the results were discussed.

Statistical word segmentation succeeds given the minimal amount of exposure.

One of the first tasks in language acquisition is word segmentation, a process to extract word forms from continuous speech streams. Statistical approaches to word segmentation have been shown to be a powerful mechanism, in which word boundaries are inferred from sequence statistics. This approach requires the learner to represent the frequency of units from syllable sequences, though accounts differ on how much statistical exposure is required. In this study, we examined the computational limit with which words can be extracted from continuous sequences. First, we discussed why two occurrences of a word in a continuous sequence is the computational lower limit for this word to be statistically defined. Next, we created short syllable sequences that contained certain words either two or four times. Learners were presented with these syllable sequences one at a time, immediately followed by a test of the novel words from these sequences. We found that, with the computationally minimal amount of two exposures, words were successfully segmented from continuous sequences. Moreover, longer syllable sequences providing four exposures to words generated more robust learning results. The implications of these results are discussed in terms of how learners segment and store the word candidates from continuous sequences.

Distribution-dependent representations in auditory category learning and generalization.

A fundamental objective in Auditory Sciences is to understand how people learn to generalize auditory category knowledge in new situations. How we generalize to novel scenarios speaks to the nature of acquired category representations and generalization mechanisms in handling perceptual variabilities and novelty. The dual learning system (DLS) framework proposes that auditory category learning involves an explicit, hypothesis-testing learning system, which is optimal for learning rule-based (RB) categories, and an implicit, procedural-based learning system, which is optimal for learning categories requiring pre-decisional information integration (II) across acoustic dimensions. Although DLS describes distinct mechanisms of two types of category learning, it is yet clear the nature of acquired representations and how we transfer them to new contexts. Here, we conducted three experiments to examine differences between II and RB category representations by examining what acoustic and perceptual novelties and variabilities affect learners' generalization success. Learners can successfully categorize different sets of untrained sounds after only eight blocks of training for both II and RB categories. The category structures and novel contexts differentially modulated the generalization success. The II learners significantly decreased generalization performances when categorizing new items derived from an untrained perceptual area and in a context with more distributed samples. In contrast, RB learners' generalizations are resistant to changes in perceptual regions but are sensitive to changes in sound dispersity. Representational similarity modeling revealed that the generalization in the more dispersed sampling context was accomplished differently by II and RB learners. II learners increased representations of perceptual similarity and decision distance to compensate for the decreased transfer of category representations, whereas the RB learners used a more computational cost strategy by default, computing the decision-bound distance to guide generalization decisions. These results suggest that distinct representations emerged after learning the two types of category structures and using different computations and flexible mechanisms in resolving generalization challenges when facing novel perceptual variability in new contexts. These findings provide new evidence for dissociated representations of auditory categories and reveal novel generalization mechanisms in resolving variabilities to maintain perceptual constancy.

Perceptual intake explains variability in statistical word segmentation.

One of the first problems in language learning is to segment words from continuous speech. Both prosodic and distributional information can be useful, and it is an important question how the two types of information are integrated. In this paper, we propose that the distinction between input (the statistical properties of the syllable sequence), and intake (how learners perceptually represent the syllable sequence) is a useful framework to integrate different sources of information. We took a novel approach, observing how a large number of syllable sequences were segmented. These sequences had the same transitional probability information for finding word boundaries but different syllables in them. We found large variability in the performance of the segmentation task, suggesting that factors other than the statistical properties of sequences were at play. This variability was explored using the input/intake asymmetry framework, which predicted that factors that shaped the representation of different syllable sequences could explain the variability of learning. We examined two factors, the saliency of the rhythm in these syllable sequences and how familiar the novel word forms in the sequence were to the existing lexicon. Both factors explained the variance in the learnability of different sequences, suggesting that processing of the sequences shaped learning. The implications of these results to computational models of statistical learning and broader implications to language learning were discussed.

Functional magnetic resonance imaging data for the neural dynamics underlying the acquisition of distinct auditory categories.

How people learn and represent auditory categories in the brain is a fundamental question in auditory neuroscience. Answering this question could provide insights into our understanding of the neurobiology of speech learning and perception. However, the neural mechanisms underlying auditory category learning are far from understood. We have revealed that the neural representations of auditory categories emerge during category training, and the type of category structures drives the emerging dynamics of the representations [1]. The dataset introduced here was derived from [1], where we collected to examine the neural dynamics underlying the acquisition of two distinct category structures: rule-based (RB) and information-integration (II) categories. Participants were trained to categorize these auditory categories with trial-by-trial corrective feedback. The functional magnetic resonance imaging (fMRI) technique was used to assess the neural dynamics related to the category learning process. Sixty adult Mandarin native speakers were recruited for the fMRI experiment. They were assigned to either the RB (n = 30, 19 females) or II (n = 30, 22 females) learning task. Each task consisted of six training blocks where each consisting of 40 trials. Spatiotemporal multivariate representational similarity analysis has been used to examine the emerging patterns of neural representations during learning [1]. This open-access dataset could potentially be reused to investigate a range of neural mechanisms (e.g., functional network organizations underlying learning of different structures of categories and neuromarkers associated with individual behavioral learning success) involved in auditory category learning.

Implicit Perception of Differences between NLP-Produced and Human-Produced Language in the Mentalizing Network.

Natural language processing (NLP) is central to the communication with machines and among ourselves, and NLP research field has long sought to produce human-quality language. Identification of informative criteria for measuring NLP-produced language quality will support development of ever-better NLP tools. The authors hypothesize that mentalizing network neural activity may be used to distinguish NLP-produced language from human-produced language, even for cases where human judges cannot subjectively distinguish the language source. Using the social chatbots Google Meena in English and Microsoft XiaoIce in Chinese to generate NLP-produced language, behavioral tests which reveal that variance of personality perceived from chatbot chats is larger than for human chats are conducted, suggesting that chatbot language usage patterns are not stable. Using an identity rating task with functional magnetic resonance imaging, neuroimaging analyses which reveal distinct patterns of brain activity in the mentalizing network including the DMPFC and rTPJ in response to chatbot versus human chats that cannot be distinguished subjectively are conducted. This study illustrates a promising empirical basis for measuring the quality of NLP-produced language: adding a judge's implicit perception as an additional criterion.

Photoelectrochemical immunosensor for HER2 detection based on BiVO4-Bi2S3 heterojunction as photoactive material and CdS as signal probe.

A sandwiched photoelectrochemical (PEC) sensor was developed for sensitive detection of human epidermal growth factor receptor 2 (HER2) based on BiVO4-Bi2S3 heterojunction as the photoelectric material accompanied with magnetic nanoparticles for enrichment of HER2 and CdS for signal amplification. The in situ generation of Bi2S3 on the surface of BiVO4 forming a BiVO4-Bi2S3 heterojunction is more conducive to the transit of electron-hole pairs. Antibody-modified MNs are utilized to capture and separate HER2 from samples. After forming a sandwich immune structure, under illumination, the photocurrent shows an increasing trend with the increment of HER2 concentration. The PEC immunosensor displays a good linear concentration range between 1.00 and 1.00 × 103 pg·mL-1 and a low limit of detection down to 0.680 pg·mL-1 (S/N = 3) for HER2 under a bias voltage of 0.1 V (vs. Ag/AgCl electrode). Furthermore, the sensor was successfully applied to detect HER2 in serum samples with recoveries that ranged between 96.1 and 114% with RSDs between 1.3 and 5.9%.

Reduced Neural Specialization for Word-level Linguistic Prosody in Children with Autism.

Children with autism often show atypical brain lateralization for speech and language processing, however, it is unclear what linguistic component contributes to this phenomenon. Here we measured event-related potential (ERP) responses in 21 school-age autistic children and 25 age-matched neurotypical (NT) peers during listening to word-level prosodic stimuli. We found that both groups displayed larger late negative response (LNR) amplitude to native prosody than to nonnative prosody; however, unlike the NT group exhibiting left-lateralized LNR distinction of prosodic phonology, the autism group showed no evidence of LNR lateralization. Moreover, in both groups, the LNR effects were only present for prosodic phonology but not for phoneme-free prosodic acoustics. These results extended the findings of inadequate neural specialization for language in autism to sub-lexical prosodic structures.

Parafoveal preview benefit in a conflicting sentential context: Evidence from ERPs.

In natural reading, the reader's processing of a word starts when the word is located in parafoveal vision. Relative to a situation with an invalid preview, fixations are significantly shorter after a fully valid, identical preview. Although research on the identity preview benefit has been ongoing for more than 40 years, the mechanism of this preview benefit, the level of lexical processing it occurs, and its relationship to the sentential context remain unclear. This study employed EEG brain component analysis technology to address these questions in Chinese sentence reading. We manipulated the sentential context to make the target word plausible or not plausible with the sentence and manipulated the target word present or not present in preview. EEG results showed that the identity preview benefit can affect not only the early preview positivity, reflecting the early orthographic processing of words, but also the N400 and LPC components, reflecting the late and in-depth semantic processing of words. Conflicting sentential context, in which the target word is implausible and cannot be integrated into the sentence, can interfere temporarily with these processes. These findings suggest that in the process of sentence reading, an identical preview word can promote the subsequent reading process at multiple levels, and its role is modulated by contextual information.

Effects of exogenous and endogenous cues on attentional orienting in deaf adults.

Adults who are deaf have been shown to have better visual attentional orienting than those with typical hearing, especially when the target is located in the periphery of the visual field. However, most studies in this population have assessed exogenous visual attention orienting (bottom-up processing of external cues) rather than endogenous visual attention orienting (top-down processing of internal cues). We used a target detection task to assess both types of visual attention orienting. A modified cue-target paradigm was adopted to assess the facilitation effects of exogenous and endogenous cues during short and long inter-stimulus intervals (ISI), using a 2 (Group: deaf/typically hearing) * 2 (Location: central/peripheral) * 2 (Cue Type: exogenous/endogenous) mixed factorial design. ANOVAs showed that both exogenous cues and endogenous cues can facilitate deaf adults' visual attentional orienting, and the facilitation effect of exogenous cues on attention orienting was significantly stronger for deaf participants than hearing participants. When the ISI was long, the effect was significantly stronger when the exogenous cue appeared in the periphery of the visual field. In the periphery, deaf adults benefited most from exogenous cues, whereas hearing adults benefited most from endogenous cues. The results suggest that not only exogenous cues but also endogenous cues can facilitate deaf adults' visual attentional orienting. However, the effect of exogenous cues appears to be greater, especially when the stimulus appears in the peripheral visual field.

Resting-state EEG reveals global network deficiency in prelingually deaf children with late cochlear implantation.

There are individual differences in rehabilitation after cochlear implantation that can be explained by brain plasticity. However, from the perspective of brain networks, the effect of implantation age on brain plasticity is unclear. The present study investigated electroencephalography functional networks in the resting state, including eyes-closed and eyes-open conditions, in 31 children with early cochlear implantation, 24 children with late cochlear implantation, and 29 children with normal hearing. Resting-state functional connectivity was measured with phase lag index, and we investigated the connectivity between the sensory regions for each frequency band. Network topology was examined using minimum spanning tree to obtain the network backbone characteristics. The results showed stronger connectivity between auditory and visual regions but reduced global network efficiency in children with late cochlear implantation in the theta and alpha bands. Significant correlations were observed between functional backbone characteristics and speech perception scores in children with cochlear implantation. Collectively, these results reveal an important effect of implantation age on the extent of brain plasticity from a network perspective and indicate that characteristics of the brain network can reflect the extent of rehabilitation of children with cochlear implantation.

Parafoveal words can modulate sentence meaning: Electrophysiological evidence from an RSVP-with-flanker task.

During natural reading, readers can take up some visual information from not-yet-fixated words to the right of the current fixation and it is well-established that this parafoveal preview facilitates the subsequent foveal processing of the word. However, the extraction and integration of word meaning from parafoveal words and their possible influence on the semantic content of the sentence are controversial. In the current study, we recorded event-related potentials (ERPs) in the RSVP-with-flankers paradigm to test whether and how updates of sentential meaning, based only on parafoveal information, may influence the subsequent foveal processing. In Chinese sentences, the congruency of parafoveal and foveal target words with the sentence was orthogonally manipulated. In contrast to previous research, we also controlled for potentially confounding effects of parafoveal-to-foveal repetition priming (identity preview effects) on the N400. Crucially, we found that the classic effect of foveal congruency on the N400 component only appeared when the word in preview had been congruent with sentence meaning; in contrast, there was no N400 as a function of foveal incongruency when the preview word had also been incongruent. These results indicate that sentence meaning rapidly adapts to parafoveal preview, altering the semantic context for the subsequently fixated word. We also show that correct parafoveal preview generally attenuates the N400 once a word is fixated, regardless of congruency. Taken together, our findings underline the highly generative and adaptive framework of language comprehension.

Generalizable predictive modeling of semantic processing ability from functional brain connectivity.

Semantic processing (SP) is one of the critical abilities of humans for representing and manipulating conceptual and meaningful information. Neuroimaging studies of SP typically collapse data from many subjects, but its neural organization and behavioral performance vary between individuals. It is not yet understood whether and how the individual variabilities in neural network organizations contribute to the individual differences in SP behaviors. We aim to identify the neural signatures underlying SP variabilities by analyzing functional connectivity (FC) patterns based on a large-sample Human Connectome Project (HCP) dataset and rigorous predictive modeling. We used a two-stage predictive modeling approach to build an internally cross-validated model and to test the model's generalizability with unseen data from different HCP samples and other out-of-sample datasets. FC patterns within a putative semantic brain network were significantly predictive of individual SP scores summarized from five SP-related behavioral tests. This cross-validated model can be used to predict unseen HCP data. The model generalizability was enhanced in the language task compared with other tasks used during scanning and was better for females than males. The model constructed from the HCP dataset can be partially generalized to two independent cohorts that participated in different semantic tasks. FCs connecting to the Perisylvian language network show the most reliable contributions to predictive modeling and the out-of-sample generalization. These findings contribute to our understanding of the neural sources of individual differences in SP, which potentially lay the foundation for personalized education for healthy individuals and intervention for SP and language deficits patients.

Word's Predictability Can Modulate Semantic Preview Effect in High-Constraint Sentences.

The processing of words in sentence reading is influenced by both information from sentential context (the effect of predictability) and information from previewing upcoming words (the preview effect), but how both effects interact during online reading is not clear. In this study, we tested the interaction of predictability effect and the preview effect in predicting reading processing. In the experiment, sentence constraint was controlled using all high-constraint sentences as materials. We manipulated both the predictability of the target word in the sentence and the semantic relationship between the preview word and the target word as predictors of the semantic preview effect. The results showed that the semantic preview effect was present only when the target word had low-predictability in the sentence but was not observed when the target word had high-predictability in the sentence. The results suggest that contextual information in reading can modulate the pre-activation of words and thus influence whether the preview word has a priming effect. The results of this study provide further evidence that reading comprehension involves an interactive system of processing multiple sources of information at multiple levels.

The role of left angular gyrus in the representation of linguistic composition relations.

Language comprehension is compositional: individual words are combined structurally to form larger meaning representations. The neural basis for compositionality is at the center of a growing body of recent research. Previous work has largely used univariate analysis to investigate the question, a technique that could potentially lead to the loss of fined-grained information due to the procedure of averaging over neural responses. In a functional magnetic resonance imaging experiment, the present study examined different types of composition relations in Chinese phrases, using a 1-back composition relation probe (CRP) task and a 1-back word probe (WP) task. We first analyzed the data using the multivariate representation similarity analysis, which better captures the fine-grained representational differences in the stimuli. The results showed that the left angular gyrus (AG) represents different types of composition relations in the CRP task, but no brain areas were identified in the WP task. We also conducted a traditional univariate analysis and found greater activations in the bilateral inferior frontal gyrus in the CRP task relative to the WP task. We discuss the methodological and theoretical implications of our findings in the context of the larger language neural network identified in previous studies. Our findings highlight the role of left AG in representing and distinguishing fine-grained linguistic composition relations.

Error-based structure prediction in language comprehension: Evidence from verb bias effects in a visual-world structural priming paradigm for Mandarin Chinese.

Structural priming studies in production have demonstrated stronger priming effects for unexpected sentence structures (inverse preference effect). This is consistent with error-based implicit learning accounts that assume learning depends on prediction error. Such prediction error can be verb-specific, leading to strong priming when a verb that is for instance biased toward the prepositional object (PO) structure occurs with an unexpected double object (DO) structure. However, it is unclear whether this mechanism also holds for language comprehension, especially for languages like Mandarin Chinese, which arguably depends strongly on semantics to predict syntax in comprehension. Experiment 1 was a norming study (N = 367) that measured the biases (DO vs. PO) of 48 Mandarin Chinese dative verbs. Experiment 2 (N = 72) crossed verb bias (DO-bias or PO-bias) and structure (DO or PO) of prime sentences in a visual-world paradigm, to examine whether Mandarin comprehenders show an inverse preference effect. The priming effect is expressed as the proportion of looks to the predicted referent (i.e., the recipient after a DO-prime, the theme after a PO-prime), for two critical time windows during target sentence processing: the verb and the first syllable of the first postverbal noun (which was identical in theme and recipient). There was priming in both time windows, even though the verb differed between prime and target. Importantly, there was an inverse preference effect (i.e., stronger priming after a DO prime with a PO-biased verb than with a DO-biased verb) in the second time window. These results provide evidence for an error-based structure prediction system in comprehension. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Phonetic Encoding Contributes to the Processing of Linguistic Prosody at the Word Level: Cross-Linguistic Evidence From Event-Related Potentials.

PURPOSE: This study aimed to examine whether abstract knowledge of word-level linguistic prosody is independent of or integrated with phonetic knowledge. METHOD: Event-related potential (ERP) responses were measured from 18 adult listeners while they listened to native and nonnative word-level prosody in speech and in nonspeech. The prosodic phonology (speech) conditions included disyllabic pseudowords spoken in Chinese and in English matched for syllabic structure, duration, and intensity. The prosodic acoustic (nonspeech) conditions were hummed versions of the speech stimuli, which eliminated the phonetic content while preserving the acoustic prosodic features. RESULTS: We observed language-specific effects on the ERP that native stimuli elicited larger late negative response (LNR) amplitude than nonnative stimuli in the prosodic phonology conditions. However, no such effect was observed in the phoneme-free prosodic acoustic control conditions. CONCLUSIONS: The results support the integration view that word-level linguistic prosody likely relies on the phonetic content where the acoustic cues embedded in. It remains to be examined whether the LNR may serve as a neural signature for language-specific processing of prosodic phonology beyond auditory processing of the critical acoustic cues at the suprasyllabic level.